{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JwrAr8vEmP2M"
   },
   "source": [
    "# Introduction\n",
    "1. This tutorial introduces *trade-based metrics* for hyperparameter optimization of FinRL models.\n",
    "2. As the name implies, trade-based metrics are associated with the trade activity that FinRL captures in its actions tables. In general, a trade is represented by an entry in an actions file.\n",
    "2. Such metrics include counts of winning and losing trades, total value of wins and losses and ratio of average market value of wins to losses.\n",
    "1. In this tutorial, we will be tuning hyperparameters for Stable baselines3 models using Optuna.\n",
    "2. The default model hyperparameters may not be adequate for your custom portfolio or custom state-space. Reinforcement learning algorithms are sensitive to hyperparameters, hence tuning is an important step.\n",
    "3. Hyperparamters are tuned based on an objective, which needs to be maximized or minimized. ***In this tutorial, the ratio of average winning to losing trade value is used as the objective.*** This ratio is to be ***maximized***.\n",
    "3. This tutorial incorporates a multi-stock framework based on the 30 stocks (aka tickers) in the DOW JONES Industrial Average. Trade metrics are calculated for each ticker and then aggregated.\n",
    "7.**IMPORTANT**: While the DOW stocks represent a portfolio, portfolio optimization techniques, such as the classic Markowitz mean-variance model, are not applied in this analysis. Other FinRL tutorials and examples demonstrate portfolio optimization. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qRF-7Vp5NCjU"
   },
   "outputs": [],
   "source": [
    "#Installing FinRL\n",
    "# Set colab status to trigger installs\n",
    "clb = True\n",
    "\n",
    "print(f'Preparing for colab: {clb}')\n",
    "pkgs = ['FinRL', 'optuna', 'Ray/rllib','plotly','ipywidgets']\n",
    "if clb:\n",
    "    print(f'Installing packages: {pkgs}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iMrTjlmamP7X"
   },
   "outputs": [],
   "source": [
    "# Set Variables\n",
    "## Fixed\n",
    "tpm_hist = {}  # record tp metric values for trials\n",
    "tp_metric = 'avgwl'  # specified trade_param_metric: ratio avg value win/loss\n",
    "## Settable by User\n",
    "n_trials = 5  # number of HP optimization runs\n",
    "total_timesteps = 2000 # per HP optimization run\n",
    "## Logging callback params\n",
    "lc_threshold=1e-5\n",
    "lc_patience=15\n",
    "lc_trial_number=5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "bSLdukRZmzHD",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%capture\n",
    "if clb:\n",
    "    # installing packages\n",
    "    !pip install pyfolio-reloaded  #original pyfolio no longer maintained\n",
    "    !pip install git+https://github.com/AI4Finance-LLC/FinRL-Library.git\n",
    "    !pip install optuna\n",
    "    !pip install -U \"ray[rllib]\"\n",
    "    !pip install plotly\n",
    "    !pip install ipywidgets\n",
    "    !pip install -U kaleido   # enables saving plots to file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ShYEIr-umzHE",
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#Importing the libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "# matplotlib.use('Agg')\n",
    "import datetime\n",
    "import optuna\n",
    "from pathlib import Path\n",
    "from google.colab import files\n",
    "%matplotlib inline\n",
    "from finrl import config\n",
    "from finrl import config_tickers\n",
    "from optuna.integration import PyTorchLightningPruningCallback\n",
    "from finrl.meta.preprocessor.yahoodownloader import YahooDownloader\n",
    "from finrl.meta.preprocessor.preprocessors import FeatureEngineer, data_split\n",
    "from finrl.meta.env_stock_trading.env_stocktrading import StockTradingEnv\n",
    "from finrl.meta.env_stock_trading.env_stocktrading_np import StockTradingEnv as StockTradingEnv_numpy\n",
    "from finrl.agents.stablebaselines3.models import DRLAgent\n",
    "from finrl.agents.rllib.models import DRLAgent as DRLAgent_rllib\n",
    "from finrl.meta.data_processor import DataProcessor\n",
    "import joblib\n",
    "from finrl.plot import backtest_stats, backtest_plot, get_daily_return, get_baseline\n",
    "import ray\n",
    "from pprint import pprint\n",
    "import kaleido\n",
    "\n",
    "import sys\n",
    "sys.path.append(\"../FinRL-Library\")\n",
    "\n",
    "import itertools\n",
    "\n",
    "import torch\n",
    "if torch.cuda.is_available():\n",
    "  device = torch.device(\"cuda\")\n",
    "else:\n",
    "  device = torch.device(\"cpu\")\n",
    "print(f'Torch device: {device}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nPE_10j0_fNV"
   },
   "source": [
    "Zipline was developed by Quantopian, which also created pyfolio. The latter is used in FinRL to calculate and display backtest results. Despite the unavailability of zipline, as reported above, pyfolio remains operational. See [here](https://github.com/quantopian/pyfolio/issues/654) for more information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "v0sD-98_8PkH"
   },
   "outputs": [],
   "source": [
    "## Connect to GPU for faster processing\n",
    "gpu_info = !nvidia-smi\n",
    "gpu_info = '\\n'.join(gpu_info)\n",
    "if gpu_info.find('failed') >= 0:\n",
    "  print('Not connected to a GPU')\n",
    "else:\n",
    "  print(gpu_info)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "z7_fCHS6NMx9"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "if not os.path.exists(\"./\" + config.DATA_SAVE_DIR):\n",
    "    os.makedirs(\"./\" + config.DATA_SAVE_DIR)\n",
    "if not os.path.exists(\"./\" + config.TRAINED_MODEL_DIR):\n",
    "    os.makedirs(\"./\" + config.TRAINED_MODEL_DIR)\n",
    "if not os.path.exists(\"./\" + config.TENSORBOARD_LOG_DIR):\n",
    "    os.makedirs(\"./\" + config.TENSORBOARD_LOG_DIR)\n",
    "if not os.path.exists(\"./\" + config.RESULTS_DIR):\n",
    "    os.makedirs(\"./\" + config.RESULTS_DIR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "71P6jMlEpikl"
   },
   "source": [
    "## Collecting data and preprocessing\n",
    "\n",
    "1.   Load DOW 30 prices\n",
    "2.   Add technical indicators\n",
    "3.   Create *processed_full*, the final data set for training and testing\n",
    "\n",
    "\n",
    "To save time in multiple runs, if the processed_full file is available, it is read from a previously saved csv file.\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "L4Vd4OHn0KcI"
   },
   "outputs": [],
   "source": [
    "#Custom ticker list dataframe download\n",
    "#TODO save df to avoid download\n",
    "path_pf = '/content/ticker_data.csv'\n",
    "if Path(path_pf).is_file():\n",
    "  print('Reading ticker data')\n",
    "  df = pd.read_csv(path_pf)\n",
    "  \n",
    "else:\n",
    "  print('Downloading ticker data')\n",
    "  ticker_list = config_tickers.DOW_30_TICKER\n",
    "  df = YahooDownloader(start_date = '2009-01-01',\n",
    "                     end_date = '2021-10-01',\n",
    "                     ticker_list = ticker_list).fetch_data()\n",
    "  df.to_csv('ticker_data.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "5diXih4zPE6m"
   },
   "outputs": [],
   "source": [
    "def create_processed_full(processed):\n",
    "  list_ticker = processed[\"tic\"].unique().tolist()\n",
    "  list_date = list(pd.date_range(processed['date'].min(),processed['date'].max()).astype(str))\n",
    "  combination = list(itertools.product(list_date,list_ticker))\n",
    "\n",
    "  processed_full = pd.DataFrame(combination,columns=[\"date\",\"tic\"]).merge(processed,on=[\"date\",\"tic\"],how=\"left\")\n",
    "  processed_full = processed_full[processed_full['date'].isin(processed['date'])]\n",
    "  processed_full = processed_full.sort_values(['date','tic'])\n",
    "\n",
    "  processed_full = processed_full.fillna(0)\n",
    "  processed_full.sort_values(['date','tic'],ignore_index=True).head(5)\n",
    "\n",
    "  processed_full.to_csv('processed_full.csv')\n",
    "  return processed_full"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "JeHXtUj4NI5R"
   },
   "outputs": [],
   "source": [
    "#You can add technical indicators and turbulence factor to dataframe\n",
    "#Just set the use_technical_indicator=True, use_vix=True and use_turbulence=True\n",
    "def create_techind():\n",
    "  fe = FeatureEngineer(\n",
    "                    use_technical_indicator=True,\n",
    "                    tech_indicator_list = config.INDICATORS,\n",
    "                    use_vix=True,\n",
    "                    use_turbulence=True,\n",
    "                    user_defined_feature = False)\n",
    "\n",
    "  processed = fe.preprocess_data(df)\n",
    "  return processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "avdupku_JcK1"
   },
   "outputs": [],
   "source": [
    "#Load price and technical indicator data from file if available\n",
    "\n",
    "path_pf = '/content/processed_full.csv'\n",
    "if Path(path_pf).is_file():\n",
    "  print('Reading processed_full data')\n",
    "  processed_full = pd.read_csv(path_pf)\n",
    "\n",
    "else:\n",
    "  print('Creating processed_full file')\n",
    "  processed=create_techind()\n",
    "  processed_full=create_processed_full(processed)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3RrJiFbSPKE2"
   },
   "outputs": [],
   "source": [
    "train = data_split(processed_full, '2009-01-01','2020-07-01')\n",
    "trade = data_split(processed_full, '2020-05-01','2021-10-01')\n",
    "print(f'Number of training samples: {len(train)}')\n",
    "print(f'Number of testing samples: {len(trade)}')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ub4JTTRcPOel"
   },
   "outputs": [],
   "source": [
    "stock_dimension = len(train.tic.unique())\n",
    "state_space = 1 + 2*stock_dimension + len(config.INDICATORS) * stock_dimension\n",
    "print(f\"Stock Dimension: {stock_dimension}, State Space: {state_space}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "YiF95zXgPTsd"
   },
   "outputs": [],
   "source": [
    "#Defining the environment kwargs\n",
    "\n",
    "env_kwargs = {\n",
    "    \"hmax\": 100, \n",
    "    \"initial_amount\": 1000000, \n",
    "    \"buy_cost_pct\": 0.001,\n",
    "    \"sell_cost_pct\": 0.001,\n",
    "    \"state_space\": state_space, \n",
    "    \"stock_dim\": stock_dimension, \n",
    "    \"tech_indicator_list\": config.INDICATORS, \n",
    "    \"action_space\": stock_dimension, \n",
    "    \"reward_scaling\": 1e-4\n",
    "    \n",
    "}\n",
    "#Instantiate the training gym compatible environment\n",
    "e_train_gym = StockTradingEnv(df = train, **env_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "892NcZALPWHF"
   },
   "outputs": [],
   "source": [
    "#Instantiate the training environment\n",
    "# Also instantiate our training gent\n",
    "env_train, _ = e_train_gym.get_sb_env()\n",
    "#print(type(env_train))\n",
    "agent = DRLAgent(env = env_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "3EwB1T8opX2o"
   },
   "outputs": [],
   "source": [
    "#Instantiate the trading environment\n",
    "e_trade_gym = StockTradingEnv(df = trade, turbulence_threshold = None, **env_kwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "id": "mPAvpM-CTJE5"
   },
   "source": [
    "# Trade performance code\n",
    "The following code calculates trade performance metrics, which are then used as an objective for optimizing hyperparameter values. \n",
    "\n",
    "There are several available metrics. In this tutorial, the default choice is the ratio of average value of winning to losing trades. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oQkEK8HvLA2S"
   },
   "outputs": [],
   "source": [
    "#Main method\n",
    "# Calculates Trade Performance for Objective\n",
    "# Called from objective method\n",
    "# Returns selected trade perf metric(s)\n",
    "# Requires actions and associated prices\n",
    "\n",
    "def calc_trade_perf_metric(df_actions, \n",
    "                           df_prices_trade,\n",
    "                           tp_metric,\n",
    "                           dbg=False):\n",
    "  \n",
    "    df_actions_p, df_prices_p, tics = prep_data(df_actions.copy(),\n",
    "                                                df_prices_trade.copy())\n",
    "    # actions predicted by trained model on trade data\n",
    "    df_actions_p.to_csv('df_actions.csv') \n",
    "\n",
    "    \n",
    "    # Confirms that actions, prices and tics are consistent\n",
    "    df_actions_s, df_prices_s, tics_prtfl = \\\n",
    "        sync_tickers(df_actions_p.copy(),df_prices_p.copy(),tics)\n",
    "    \n",
    "    # copy to ensure that tics from portfolio remains unchanged\n",
    "    tics = tics_prtfl.copy()\n",
    "    \n",
    "    # Analysis is performed on each portfolio ticker\n",
    "    perf_data= collect_performance_data(df_actions_s, df_prices_s, tics)\n",
    "    # profit/loss for each ticker\n",
    "    pnl_all = calc_pnl_all(perf_data, tics)\n",
    "    # values for trade performance metrics\n",
    "    perf_results = calc_trade_perf(pnl_all)\n",
    "    df = pd.DataFrame.from_dict(perf_results, orient='index')\n",
    "    \n",
    "    # calculate and return trade metric value as objective\n",
    "    m = calc_trade_metric(df,tp_metric)\n",
    "    print(f'Ratio Avg Win/Avg Loss: {m}')\n",
    "    k = str(len(tpm_hist)+1)\n",
    "    # save metric value\n",
    "    tpm_hist[k] = m\n",
    "    return m\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "GJipWp6UByzi"
   },
   "outputs": [],
   "source": [
    "# Supporting methods\n",
    "def calc_trade_metric(df,metric='avgwl'):\n",
    "    '''# trades', '# wins', '# losses', 'wins total value', 'wins avg value',\n",
    "       'losses total value', 'losses avg value'''\n",
    "    # For this tutorial, the only metric available is the ratio of \n",
    "    #  average values of winning to losing trades. Others are in development.\n",
    "    \n",
    "    # some test cases produce no losing trades.\n",
    "    # The code below assigns a value as a multiple of the highest value during\n",
    "    # previous hp optimization runs. If the first run experiences no losses,\n",
    "    # a fixed value is assigned for the ratio\n",
    "    tpm_mult = 1.0\n",
    "    avgwl_no_losses = 25\n",
    "    if metric == 'avgwl':\n",
    "        if sum(df['# losses']) == 0:\n",
    "          try:\n",
    "            return max(tpm_hist.values())*tpm_mult\n",
    "          except ValueError:\n",
    "            return avgwl_no_losses\n",
    "        avg_w = sum(df['wins total value'])/sum(df['# wins'])\n",
    "        avg_l = sum(df['losses total value'])/sum(df['# losses'])\n",
    "        m = abs(avg_w/avg_l)\n",
    "\n",
    "    return m\n",
    "\n",
    "\n",
    "def prep_data(df_actions,\n",
    "              df_prices_trade):\n",
    "    \n",
    "    df=df_prices_trade[['date','close','tic']]\n",
    "    df['Date'] = pd.to_datetime(df['date'])\n",
    "    df = df.set_index('Date')\n",
    "    # set indices on both df to datetime\n",
    "    idx = pd.to_datetime(df_actions.index, infer_datetime_format=True)\n",
    "    df_actions.index=idx\n",
    "    tics = np.unique(df.tic)\n",
    "    n_tics = len(tics)\n",
    "    print(f'Number of tickers: {n_tics}')\n",
    "    print(f'Tickers: {tics}')\n",
    "    dategr = df.groupby('tic')\n",
    "    p_d={t:dategr.get_group(t).loc[:,'close'] for t in tics}\n",
    "    df_prices = pd.DataFrame.from_dict(p_d)\n",
    "    df_prices.index = df_prices.index.normalize()\n",
    "    return df_actions, df_prices, tics\n",
    "\n",
    "\n",
    "# prepares for integrating action and price files\n",
    "def link_prices_actions(df_a,\n",
    "                        df_p):\n",
    "    cols_a = [t + '_a' for t in df_a.columns]\n",
    "    df_a.columns = cols_a\n",
    "    cols_p = [t + '_p' for t in df_p.columns]\n",
    "    df_p.columns = cols_p\n",
    "    return df_a, df_p\n",
    "\n",
    "\n",
    "def sync_tickers(df_actions,df_tickers_p,tickers):\n",
    "    # Some DOW30 components may not be included in portfolio\n",
    "    # passed tickers includes all DOW30 components\n",
    "    # actions and ticker files may have different length indices\n",
    "    if len(df_actions) != len(df_tickers_p):\n",
    "      msng_dates = set(df_actions.index)^set(df_tickers_p.index)\n",
    "      try:\n",
    "        #assumption is prices has one additional timestamp (row)\n",
    "        df_tickers_p.drop(msng_dates,inplace=True)\n",
    "      except:\n",
    "        df_actions.drop(msng_dates,inplace=True)\n",
    "    df_actions, df_tickers_p = link_prices_actions(df_actions,df_tickers_p)\n",
    "    # identify any DOW components not in portfolio\n",
    "    t_not_in_a = [t for t in tickers if t + '_a' not in list(df_actions.columns)]\n",
    "  \n",
    "    # remove t_not_in_a from df_tickers_p\n",
    "    drop_cols = [t + '_p' for t in t_not_in_a]\n",
    "    df_tickers_p.drop(columns=drop_cols,inplace=True)\n",
    "    \n",
    "    # Tickers in portfolio\n",
    "    tickers_prtfl = [c.split('_')[0] for c in df_actions.columns]\n",
    "    return df_actions,df_tickers_p, tickers_prtfl\n",
    "\n",
    "def collect_performance_data(dfa,dfp,tics, dbg=False):\n",
    "    \n",
    "    perf_data = {}\n",
    "    # In current version, files columns include secondary identifier\n",
    "    for t in tics:\n",
    "        # actions: purchase/sale of DOW equities\n",
    "        acts = dfa['_'.join([t,'a'])].values\n",
    "        # ticker prices\n",
    "        prices = dfp['_'.join([t,'p'])].values\n",
    "        # market value of purchases/sales\n",
    "        tvals_init = np.multiply(acts,prices)\n",
    "        d={'actions':acts, 'prices':prices,'init_values':tvals_init}\n",
    "        perf_data[t]=d\n",
    "\n",
    "    return perf_data\n",
    "\n",
    "\n",
    "def calc_pnl_all(perf_dict, tics_all):\n",
    "    # calculate profit/loss for each ticker\n",
    "    print(f'Calculating profit/loss for each ticker')\n",
    "    pnl_all = {}\n",
    "    for tic in tics_all:\n",
    "        pnl_t = []\n",
    "        tic_data = perf_dict[tic]\n",
    "        init_values = tic_data['init_values']\n",
    "        acts = tic_data['actions']\n",
    "        prices = tic_data['prices']\n",
    "        cs = np.cumsum(acts)\n",
    "        args_s = [i + 1 for i in range(len(cs) - 1) if cs[i + 1] < cs[i]]\n",
    "        # tic actions with no sales\n",
    "        if not args_s:\n",
    "            pnl = complete_calc_buyonly(acts, prices, init_values)\n",
    "            pnl_all[tic] = pnl\n",
    "            continue\n",
    "        # copy acts: acts_rev will be revised based on closing/reducing init positions\n",
    "        pnl_all = execute_position_sales(tic,acts,prices,args_s,pnl_all)\n",
    "\n",
    "    return pnl_all\n",
    "\n",
    "\n",
    "def complete_calc_buyonly(actions, prices, init_values):\n",
    "    # calculate final pnl for each ticker assuming no sales\n",
    "    fnl_price = prices[-1]\n",
    "    final_values = np.multiply(fnl_price, actions)\n",
    "    pnl = np.subtract(final_values, init_values)\n",
    "    return pnl\n",
    "\n",
    "\n",
    "def execute_position_sales(tic,acts,prices,args_s,pnl_all):\n",
    "  # calculate final pnl for each ticker with sales\n",
    "    pnl_t = []\n",
    "    acts_rev = acts.copy()\n",
    "    # location of sales transactions\n",
    "    for s in args_s:  # s is scaler\n",
    "        # price_s = [prices[s]]\n",
    "        act_s = [acts_rev[s]]\n",
    "        args_b = [i for i in range(s) if acts_rev[i] > 0]\n",
    "        prcs_init_trades = prices[args_b]\n",
    "        acts_init_trades = acts_rev[args_b]\n",
    "  \n",
    "        # update actions for sales\n",
    "        # reduce/eliminate init values through trades\n",
    "        # always start with earliest purchase that has not been closed through sale\n",
    "        # selectors for purchase and sales trades\n",
    "        # find earliest remaining purchase\n",
    "        arg_sel = min(args_b)\n",
    "        # sel_s = len(acts_trades) - 1\n",
    "\n",
    "        # closing part/all of earliest init trade not yet closed\n",
    "        # sales actions are negative\n",
    "        # in this test case, abs_val of init and sales share counts are same\n",
    "        # zero-out sales actions\n",
    "        # market value of sale\n",
    "        # max number of shares to be closed: may be less than # originally purchased\n",
    "        acts_shares = min(abs(act_s.pop()), acts_rev[arg_sel])\n",
    "\n",
    "        # mv of shares when purchased\n",
    "        mv_p = abs(acts_shares * prices[arg_sel])\n",
    "        # mv of sold shares\n",
    "        mv_s = abs(acts_shares * prices[s])\n",
    "\n",
    "        # calc pnl\n",
    "        pnl = mv_s - mv_p\n",
    "        # reduce init share count\n",
    "        # close all/part of init purchase\n",
    "        acts_rev[arg_sel] -= acts_shares\n",
    "        acts_rev[s] += acts_shares\n",
    "        # calculate pnl for trade\n",
    "        # value of associated purchase\n",
    "        \n",
    "        # find earliest non-zero positive act in acts_revs\n",
    "        pnl_t.append(pnl)\n",
    "    \n",
    "    pnl_op = calc_pnl_for_open_positions(acts_rev, prices)\n",
    "    #pnl_op is list\n",
    "    # add pnl_op results (if any) to pnl_t (both lists)\n",
    "    pnl_t.extend(pnl_op)\n",
    "    #print(f'Total pnl for {tic}: {np.sum(pnl_t)}')\n",
    "    pnl_all[tic] = np.array(pnl_t)\n",
    "    return pnl_all\n",
    "\n",
    "\n",
    "def calc_pnl_for_open_positions(acts,prices):\n",
    "    # identify any positive share values after accounting for sales\n",
    "    pnl = []\n",
    "    fp = prices[-1] # last price\n",
    "    open_pos_arg = np.argwhere(acts>0)\n",
    "    if len(open_pos_arg)==0:return pnl # no open positions\n",
    "\n",
    "    mkt_vals_open = np.multiply(acts[open_pos_arg], prices[open_pos_arg])\n",
    "    # mkt val at end of testing period\n",
    "    # treat as trades for purposes of calculating pnl at end of testing period\n",
    "    mkt_vals_final = np.multiply(fp, acts[open_pos_arg])\n",
    "    pnl_a = np.subtract(mkt_vals_final, mkt_vals_open)\n",
    "    #convert to list\n",
    "    pnl = [i[0] for i in pnl_a.tolist()]\n",
    "    #print(f'Market value of open positions at end of testing {pnl}')\n",
    "    return pnl\n",
    "\n",
    "\n",
    "def calc_trade_perf(pnl_d):\n",
    "    # calculate trade performance metrics\n",
    "    perf_results = {}\n",
    "    for t,pnl in pnl_d.items():\n",
    "        wins = pnl[pnl>0]  # total val\n",
    "        losses = pnl[pnl<0]\n",
    "        n_wins = len(wins)\n",
    "        n_losses = len(losses)\n",
    "        n_trades = n_wins + n_losses\n",
    "        wins_val = np.sum(wins)\n",
    "        losses_val = np.sum(losses)\n",
    "        wins_avg = 0 if n_wins==0 else np.mean(wins)\n",
    "        #print(f'{t} n_wins: {n_wins} n_losses: {n_losses}')\n",
    "        losses_avg = 0 if n_losses==0 else np.mean(losses)\n",
    "        d = {'# trades':n_trades,'# wins':n_wins,'# losses':n_losses,\n",
    "             'wins total value':wins_val, 'wins avg value':wins_avg,\n",
    "             'losses total value':losses_val, 'losses avg value':losses_avg,}\n",
    "        perf_results[t] = d\n",
    "    return perf_results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "zOynTQluppye"
   },
   "source": [
    "## Tuning hyperparameters using Optuna\n",
    "1. Go to this [link](https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/utils/hyperparams_opt.py), you will find all possible hyperparamters to tune for all the models.\n",
    "2. For your model, grab those hyperparamters which you want to optimize and then return a dictionary of hyperparamters.\n",
    "3. There is a feature in Optuna called as hyperparamters importance, you can point out those hyperparamters which are important for tuning.\n",
    "4. By default Optuna use [TPESampler](https://www.youtube.com/watch?v=tdwgR1AqQ8Y) for sampling hyperparameters from the search space. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_vojRvAsP2ja"
   },
   "outputs": [],
   "source": [
    "def sample_ddpg_params(trial:optuna.Trial):\n",
    "  # Size of the replay buffer\n",
    "  buffer_size = trial.suggest_categorical(\"buffer_size\", [int(1e4), int(1e5), int(1e6)])\n",
    "  learning_rate = trial.suggest_loguniform(\"learning_rate\", 1e-5, 1)\n",
    "  batch_size = trial.suggest_categorical(\"batch_size\", [32, 64, 128, 256, 512])\n",
    "  \n",
    "  return {\"buffer_size\": buffer_size,\n",
    "          \"learning_rate\":learning_rate,\n",
    "          \"batch_size\":batch_size}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TWtfSG9LNFXb"
   },
   "source": [
    "*OPTIONAL CODE FOR SAMPLING HYPERPARAMETERS*\n",
    "\n",
    "Replace current call in function *objective* with \n",
    "\n",
    "`hyperparameters = sample_ddpg_params_all(trial)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0MIVEjmxhwOO"
   },
   "outputs": [],
   "source": [
    "def sample_ddpg_params_all(trial:optuna.Trial,\n",
    "                           # fixed values from previous study\n",
    "                           learning_rate=0.0103,\n",
    "                           batch_size=128,\n",
    "                           buffer_size=int(1e6)):\n",
    "\n",
    "    gamma = trial.suggest_categorical(\"gamma\", [0.94, 0.96, 0.98])\n",
    "    # Polyak coeff\n",
    "    tau = trial.suggest_categorical(\"tau\", [0.08, 0.1, 0.12])\n",
    "\n",
    "    train_freq = trial.suggest_categorical(\"train_freq\", [512,768,1024])\n",
    "    gradient_steps = train_freq\n",
    "    \n",
    "    noise_type = trial.suggest_categorical(\"noise_type\", [\"ornstein-uhlenbeck\", \"normal\", None])\n",
    "    noise_std = trial.suggest_categorical(\"noise_std\", [.1,.2,.3] )\n",
    "\n",
    "    # NOTE: Add \"verybig\" to net_arch when tuning HER (see TD3)\n",
    "    net_arch = trial.suggest_categorical(\"net_arch\", [\"small\", \"big\"])\n",
    "    # activation_fn = trial.suggest_categorical('activation_fn', [nn.Tanh, nn.ReLU, nn.ELU, nn.LeakyReLU])\n",
    "\n",
    "    net_arch = {\n",
    "        \"small\": [64, 64],\n",
    "        \"medium\": [256, 256],\n",
    "        \"big\": [512, 512],\n",
    "    }[net_arch]\n",
    "  \n",
    "    hyperparams = {\n",
    "        \"batch_size\": batch_size,\n",
    "        \"buffer_size\": buffer_size,\n",
    "        \"gamma\": gamma,\n",
    "        \"gradient_steps\": gradient_steps,\n",
    "        \"learning_rate\": learning_rate,\n",
    "        \"tau\": tau,\n",
    "        \"train_freq\": train_freq,\n",
    "        #\"noise_std\": noise_std,\n",
    "        #\"noise_type\": noise_type,\n",
    "        \n",
    "        \"policy_kwargs\": dict(net_arch=net_arch)\n",
    "    }\n",
    "    return hyperparams\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gCRy_kL648DM"
   },
   "source": [
    "## Callbacks\n",
    "1. The callback will terminate if the improvement margin is below certain point\n",
    "2. It will terminate after certain number of trial_number are reached, not before that\n",
    "3. It will hold its patience to reach the threshold"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "MTPSF3bpHDUT"
   },
   "outputs": [],
   "source": [
    "class LoggingCallback:\n",
    "    def __init__(self,threshold,trial_number,patience):\n",
    "      '''\n",
    "      threshold:int tolerance for increase in objective\n",
    "      trial_number: int Prune after minimum number of trials\n",
    "      patience: int patience for the threshold\n",
    "      '''\n",
    "      self.threshold = threshold\n",
    "      self.trial_number  = trial_number\n",
    "      self.patience = patience\n",
    "      print(f'Callback threshold {self.threshold}, \\\n",
    "            trial_number {self.trial_number}, \\\n",
    "            patience {self.patience}')\n",
    "      self.cb_list = [] #Trials list for which threshold is reached\n",
    "    def __call__(self,study:optuna.study, frozen_trial:optuna.Trial):\n",
    "      #Setting the best value in the current trial\n",
    "      study.set_user_attr(\"previous_best_value\", study.best_value)\n",
    "      \n",
    "      #Checking if the minimum number of trials have pass\n",
    "      if frozen_trial.number >self.trial_number:\n",
    "          previous_best_value = study.user_attrs.get(\"previous_best_value\",None)\n",
    "          #Checking if the previous and current objective values have the same sign\n",
    "          if previous_best_value * study.best_value >=0:\n",
    "              #Checking for the threshold condition\n",
    "              if abs(previous_best_value-study.best_value) < self.threshold: \n",
    "                  self.cb_list.append(frozen_trial.number)\n",
    "                  #If threshold is achieved for the patience amount of time\n",
    "                  if len(self.cb_list)>self.patience:\n",
    "                      print('The study stops now...')\n",
    "                      print('With number',frozen_trial.number ,'and value ',frozen_trial.value)\n",
    "                      print('The previous and current best values are {} and {} respectively'\n",
    "                              .format(previous_best_value, study.best_value))\n",
    "                      study.stop()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QsEKc-9APaS1"
   },
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "import sys   \n",
    "\n",
    "os.makedirs(\"models\",exist_ok=True)\n",
    "\n",
    "def objective(trial:optuna.Trial):\n",
    "  #Trial will suggest a set of hyperparamters from the specified range\n",
    "\n",
    "  # Optional to optimize larger set of parameters\n",
    "  # hyperparameters = sample_ddpg_params_all(trial)\n",
    "  \n",
    "  # Optimize buffer size, batch size, learning rate\n",
    "  hyperparameters = sample_ddpg_params(trial)\n",
    "  #print(f'Hyperparameters from objective: {hyperparameters.keys()}')\n",
    "  policy_kwargs = None  # default\n",
    "  if 'policy_kwargs' in hyperparameters.keys():\n",
    "    policy_kwargs = hyperparameters['policy_kwargs']\n",
    "    del hyperparameters['policy_kwargs']\n",
    "    #print(f'Policy keyword arguments {policy_kwargs}')\n",
    "  model_ddpg = agent.get_model(\"ddpg\",\n",
    "                               policy_kwargs = policy_kwargs,\n",
    "                               model_kwargs = hyperparameters )\n",
    "  #You can increase it for better comparison\n",
    "  trained_ddpg = agent.train_model(model=model_ddpg,\n",
    "                                   tb_log_name=\"ddpg\",\n",
    "                                   total_timesteps=total_timesteps)\n",
    "  trained_ddpg.save('models/ddpg_{}.pth'.format(trial.number))\n",
    "  clear_output(wait=True)\n",
    "  #For the given hyperparamters, determine the account value in the trading period\n",
    "  df_account_value, df_actions = DRLAgent.DRL_prediction(\n",
    "    model=trained_ddpg, \n",
    "    environment = e_trade_gym)\n",
    " \n",
    "  # Calculate trade performance metric\n",
    "  # Currently ratio of average win and loss market values\n",
    "  tpm = calc_trade_perf_metric(df_actions,trade,tp_metric)\n",
    "  return tpm\n",
    "\n",
    "#Create a study object and specify the direction as 'maximize'\n",
    "#As you want to maximize sharpe\n",
    "#Pruner stops not promising iterations\n",
    "#Use a pruner, else you will get error related to divergence of model\n",
    "#You can also use Multivariate samplere\n",
    "#sampler = optuna.samplers.TPESampler(multivarite=True,seed=42)\n",
    "sampler = optuna.samplers.TPESampler()\n",
    "study = optuna.create_study(study_name=\"ddpg_study\",direction='maximize',\n",
    "                            sampler = sampler, pruner=optuna.pruners.HyperbandPruner())\n",
    "\n",
    "logging_callback = LoggingCallback(threshold=lc_threshold,\n",
    "                                   patience=lc_patience,\n",
    "                                   trial_number=lc_trial_number)\n",
    "#You can increase the n_trials for a better search space scanning\n",
    "study.optimize(objective, n_trials=n_trials,catch=(ValueError,),callbacks=[logging_callback])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "sg7LRlHmj9GB"
   },
   "outputs": [],
   "source": [
    "joblib.dump(study, \"final_ddpg_study__.pkl\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "SVMue1-xuGHC"
   },
   "outputs": [],
   "source": [
    "#Get the best hyperparamters\n",
    "print('Hyperparameters after tuning',study.best_params)\n",
    "print('Hyperparameters before tuning',config.DDPG_PARAMS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tsZmMw0ykmYo"
   },
   "outputs": [],
   "source": [
    "study.best_trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "fETqKJj4uSi5"
   },
   "outputs": [],
   "source": [
    "from stable_baselines3 import DDPG\n",
    "tuned_model_ddpg = DDPG.load('models/ddpg_{}.pth'.format(study.best_trial.number),env=env_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FgchX1LLuua-"
   },
   "outputs": [],
   "source": [
    "#Trading period account value with tuned model\n",
    "df_account_value_tuned, df_actions_tuned = DRLAgent.DRL_prediction(\n",
    "    model=tuned_model_ddpg, \n",
    "    environment = e_trade_gym)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9LBNBB-gyiz9"
   },
   "outputs": [],
   "source": [
    "def add_trade_perf_metric(df_actions, \n",
    "                          perf_stats_all,\n",
    "                          trade, \n",
    "                          tp_metric):\n",
    "  tpm = calc_trade_perf_metric(df_actions,trade,tp_metric)\n",
    "  trp_metric = {'Value':tpm}\n",
    "  df2 = pd.DataFrame(trp_metric,index=['Trade_Perf'])\n",
    "  perf_stats_all = perf_stats_all.append(df2)\n",
    "  return perf_stats_all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-blnTnoI1F65"
   },
   "outputs": [],
   "source": [
    "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
    "df_actions_tuned.to_csv(\"./\"+config.RESULTS_DIR+\"/tuned_actions_\" +now+ '.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8s5KNvVuvr2D"
   },
   "outputs": [],
   "source": [
    "#Backtesting with our pruned model\n",
    "print(\"==============Get Backtest Results===========\")\n",
    "print(\"==============Pruned Model===========\")\n",
    "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
    "\n",
    "perf_stats_all_tuned = backtest_stats(account_value=df_account_value_tuned)\n",
    "perf_stats_all_tuned = pd.DataFrame(perf_stats_all_tuned)\n",
    "perf_stats_all_tuned.columns = ['Value']\n",
    "# add trade performance metric\n",
    "perf_stats_all_tuned = \\\n",
    "  add_trade_perf_metric(df_actions_tuned,\n",
    "                        perf_stats_all_tuned,\n",
    "                        trade,\n",
    "                        tp_metric)\n",
    "perf_stats_all_tuned.to_csv(\"./\"+config.RESULTS_DIR+\"/perf_stats_all_tuned_\"+now+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "tuGaI9lSvvVD"
   },
   "outputs": [],
   "source": [
    "#Now train with not tuned hyperaparameters\n",
    "#Default config.ddpg_PARAMS\n",
    "non_tuned_model_ddpg = agent.get_model(\"ddpg\",model_kwargs = config.DDPG_PARAMS )\n",
    "trained_ddpg = agent.train_model(model=non_tuned_model_ddpg, \n",
    "                             tb_log_name='ddpg',\n",
    "                             total_timesteps=total_timesteps)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yZbEYRQ1wBeC"
   },
   "outputs": [],
   "source": [
    "df_account_value, df_actions = DRLAgent.DRL_prediction(\n",
    "    model=trained_ddpg, \n",
    "    environment = e_trade_gym)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pFdB4YM3wh0m"
   },
   "outputs": [],
   "source": [
    "#Backtesting for not tuned hyperparamters\n",
    "print(\"==============Get Backtest Results===========\")\n",
    "print(\"============Default Hyperparameters==========\")\n",
    "now = datetime.datetime.now().strftime('%Y%m%d-%Hh%M')\n",
    "\n",
    "perf_stats_all = backtest_stats(account_value=df_account_value)\n",
    "perf_stats_all = pd.DataFrame(perf_stats_all)\n",
    "perf_stats_all.columns = ['Value']\n",
    "# add trade performance metric\n",
    "perf_stats_all = add_trade_perf_metric(df_actions,\n",
    "                        perf_stats_all,\n",
    "                        trade,\n",
    "                        tp_metric)\n",
    "perf_stats_all.to_csv(\"./\"+config.RESULTS_DIR+\"/perf_stats_all_\"+now+'.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "12fxdvUZwi_W"
   },
   "outputs": [],
   "source": [
    "#Certainly you can afford more number of trials for further optimization\n",
    "from optuna.visualization import plot_optimization_history\n",
    "fig = plot_optimization_history(study)\n",
    "#\"./\"+config.RESULTS_DIR+\n",
    "fig.write_image(\"./\"+config.RESULTS_DIR+\"/opt_hist.png\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_TUF2GvAx6-k"
   },
   "outputs": [],
   "source": [
    "from optuna.visualization import plot_contour\n",
    "from optuna.visualization import plot_edf\n",
    "from optuna.visualization import plot_intermediate_values\n",
    "from optuna.visualization import plot_optimization_history\n",
    "from optuna.visualization import plot_parallel_coordinate\n",
    "from optuna.visualization import plot_param_importances\n",
    "from optuna.visualization import plot_slice"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2jkqeSUIyCT0"
   },
   "outputs": [],
   "source": [
    "#Hyperparamters importance\n",
    "\n",
    "try:\n",
    "  fig = plot_param_importances(study)\n",
    "  fig.write_image(\"./\"+config.RESULTS_DIR+\"/params_importances.png\")\n",
    "  fig.show()\n",
    "except:\n",
    "  print('Cannot calculate hyperparameter importances: no variation')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "_edBJqB8yEDr"
   },
   "outputs": [],
   "source": [
    "fig = plot_edf(study)\n",
    "fig.write_image(\"./\"+config.RESULTS_DIR+\"/emp_dist_func.png\")\n",
    "fig.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "pXTgG1yvgeyq"
   },
   "outputs": [],
   "source": [
    "files.download('/content/final_ddpg_study__.pkl')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "fAD0MIAWukB9"
   },
   "source": [
    "## Further Works\n",
    "\n",
    "1.   You can tune more critical hyperparameters\n",
    "2.   Multi-objective hyperparameter optimization using Optuna. Here we can maximize a trade performance metric and simultaneously minimize volatility in our account value to tune our hyperparameters\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "machine_shape": "hm",
   "name": "FinRL_HP_Tuning_Trade_Performance_with_Optuna_AI4F .ipynb",
   "private_outputs": true,
   "provenance": [
    {
     "file_id": "1NJO7xZNBX3mJs5i0vqxzReZt0P_4AA9j",
     "timestamp": 1644075428812
    },
    {
     "file_id": "1mgtB5m3ZEmW5ri8d7xXXWgDVM5qqdAYw",
     "timestamp": 1643837987521
    },
    {
     "file_id": "1JpaNsWz4RayOKqnBs1_dLv4PfF-5M1fQ",
     "timestamp": 1642777369114
    }
   ]
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
